Profiling and Tracing in Linux

Profiling and tracing tools can help make application parallelization more effective and identify performance bottlenecks. Profiling presents summary statistics of performance metrics while tracing highlights the temporal aspect of performance variations, showing when and where in the code performance is achieved. A complex challenge is the mapping of performance data gathered during execution to high-level parallel language constructs in the application source code. Presenting performance data in a meaningful way to the user is equally important. This paper presents a brief overview of profiling and tracing tools in the context of Linux the operating system most commonly used to build clusters of workstations for high performance computing. back to source-code constructs and entities that the user can comprehend is important. As shown in Figure 1., instrumentation can be added to the program at any stage during the compilation/execution process from user-added calls in the source code to a completely automated approach. (Events are triggered by the execution of the instrumentation instructions at runtime.) Moving down the stages from source code instrumentation to runtime instrumentation, the instrumentation mechanism changes from language specific to platform specific. Making an instrumentation API specific to a language for source code instrumentation may make the instrumentation portable across multiple compilers and platforms that support the language. However, the instrumentation may not work with other languages. Targeting the instrumentation at the executable level ensures that applications written in any language that generates an executable would benefit from the instrumentation on the specific platform. However, it is more difficult to map performance data back to source-code entities, especially if the application’s highlevel programming language (such as HPF or ZPL), is translated to an intermediate language (such as Fortran or C) during compilation. There is often a trade-off between what level of abstraction the tool can provide and how easily the instrumentation can be added. The JEWEL [19] package requires that calls be added manually to the source code. While providing flexibility, the process can be cumbersome and time consuming. A preprocessor can generate instrumented source code by automatically inserting annotations into the original source. This approach is taken by SvPablo [14] and AIMS [20] for C and Fortran, and by TAU [15] with PDT for C++ [18]. Instrumentation can be added by a compiler as in gprof [5] (with the -pg commandline switch for the GNU compilers). Instrumentation can also be added during linking when the executable image is created by linking multiple object files and libraries together. Packages that use this method include VampirTrace [13], and PICL [12] for MPI and PVM message passing libraries. When an inter-process communication call is executed, the instrumented library acts as a wrapper that calls the profiling and tracing routines before and after calling the corresponding uninstrumented library call. This is achieved by weak library bindings. However, often program sources and libraries are unavailable to the users and it is necessary to examine the performance of executable images. Pixie [16] and Atom [17] are tools that rewrite binary executable files while adding the instrumentation, but are not currently available under Linux. All these approaches require the programer to modify the application, re-compile and re-execute the instrumented application to produce the performance data. Sometimes, re-execution of an application using the performance analysis tool is not a viable alternative while searching for performance bottlenecks. This is true for long-running tasks such as database servers. Paradyne [9] automates the search for bottlenecks in an application at runtime by selectively inserting and deleting instrumentation using the DynInst [7] dynamic instrumentation package. After instrumentation, we execute the program and perform measurements, in the form of profiling and tracing and analyze the data.